Information processing apparatus, control method, and program

ABSTRACT

An information processing apparatus (2000) includes a detection unit (2020), a state estimation unit (2040), and a height estimation unit (2080). The detection unit (2020) detects a target person from a video frame. The state estimation unit (2040) estimates a state of the detected target person. The height estimation unit (2080) estimates a height of the person on the basis of a height of the target person in the video frame in a case where the estimated state satisfies a predetermined condition.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 17/690,140 filed on Mar. 9, 2022, which is acontinuation application of U.S. patent application Ser. No. 16/650,544filed on Mar. 25, 2020, which issued as U.S. Pat. No. 11,308,315, whichis a National Stage Entry of international applicationPCT/JP2017/034940, filed on Sep. 27, 2017, the disclosures of all ofwhich are incorporated in their entirety by reference herein.

TECHNICAL FIELD

The present invention relates to image processing.

BACKGROUND ART

A technology for tracking a person who is included in a video generatedby a camera has been developed. Patent Documents 1 and 2 are documentsdisclosing such a technology. In Patent Document 1, a method ofclassifying a state of a person into “walking”, “upright”, “sitting”,and “lying” and estimating a foot position in a real space is disclosed.In this method, after the person is detected from an image, the threestates “standing”, “sitting”, and “lying” are distinguished by combiningpose determination based on an inclination of a principal axis of aperson region with pose determination using a horizontal distance fromthe camera to a head part and a horizontal distance from the camera tothe foot. At this point, the horizontal distance from the camera to thehead part is computed by converting coordinates of the head part in animage into coordinates in the real space using an assumed height of thehead part and computing a difference between an obtained position and aposition of the camera. In the state “standing”, computation isperformed by assuming that the height of the head part is a body heightvalue. In the states “sitting” and “lying”, a predetermined value isused.

In addition, in Patent Document 1, a standstill state or a walking stateis determined by comparing positions between frames at a certain timeinterval. Finally, any of the states “walking”, “upright”, “sitting”,and “lying” is determined by combining the determined state with a posestate described above. Coordinates of the person in the real space arecomputed using a camera parameter depending on the obtained state. Inaddition, a trajectory of the person is smoothed by interpolationbetween states, and trajectory information of the person is computed.

Patent Document 2 discloses a method of specifying an area in which thefoot is seen from the camera as a body height identification region, andcomputing a body height by converting positions of the foot and the headpart of the person in the image into coordinates in the real space. Theheight identification region is defined in a position at which the footto the head part can be imaged by the camera. When the person entersthis region, the position of the foot and the position of the head arecomputed from the image, and the body height is computed.

RELATED DOCUMENT Patent Document

[Patent Document 1] Japanese Patent Application Publication No.2002-197463

[Patent Document 2] Japanese Patent Application Publication No.2001-056853

SUMMARY OF THE INVENTION Technical Problem

In Patent Document 1, a method of computing the body height of theperson included in the image and a method of computing the height of thehead part in the state “sitting” or “lying” are not mentioned. In PatentDocument 2, a pose of the person in the body height identificationregion is not considered. Thus, for example, in a case where the personis bending in the body height identification region, the body height ofthe person cannot be correctly computed. In addition, since the bodyheight of the person can be estimated only when the person is includedin the body height identification region, the body height cannot beestimated in a case where persons overlap with each other in this regionin the image or the foot of the person cannot be detected from theimage.

The present invention is conceived in view of the above problem. Oneobject of the present invention is to provide a technology forestimating a body height of a person included in an image generated by acamera with high accuracy.

Solution to Problem

An information processing apparatus of the present invention includes 1)a detection unit that detects a person from a video frame, 2) a stateestimation unit that estimates a state of a target person using a resultof the detection, and 3) a body height estimation unit that estimates abody height of the target person on the basis of a height of the targetperson in the video frame in a case where the state of the target personsatisfies a predetermined condition.

A control method of the present invention is executed by a computer. Thecontrol method includes 1) a detection step of detecting a person from avideo frame, 2) a state estimation step of estimating a state of atarget person using a result of the detection, and 3) a body heightestimation step of estimating a body height of the target person on thebasis of a height of the target person in the video frame in a casewhere the state of the target person satisfies a predeterminedcondition.

A program of the present invention causes a computer to execute eachstep of the control method of the present invention.

Advantageous Effects of Invention

According to the present invention, a technology for estimating a bodyheight of a person included in an image generated by a camera with highaccuracy is provided.

BRIEF DESCRIPTION OF THE DRAWINGS

The above object and other objects, features, and advantages will becomemore apparent from exemplary example embodiments set forth below and thefollowing drawings appended thereto.

FIG. 1 is a diagram for describing a summary of operation of aninformation processing apparatus of Example Embodiment 1.

FIG. 2 is a diagram illustrating a video frame in which a target personis included.

FIG. 3 is a diagram illustrating a configuration of the informationprocessing apparatus of Example Embodiment 1.

FIG. 4 is a diagram illustrating a computer for implementing theinformation processing apparatus.

FIG. 5 is a first flowchart illustrating a flow of process executed bythe information processing apparatus of Example Embodiment 1.

FIG. 6 is a second flowchart illustrating a flow of process executed bythe information processing apparatus of Example Embodiment 1.

FIG. 7 is a diagram illustrating tracking information.

FIG. 8 is a diagram illustrating a functional configuration of aninformation processing apparatus of Example Embodiment 2.

FIG. 9 is a first flowchart illustrating a flow of process executed bythe information processing apparatus of Example Embodiment 2.

FIG. 10 is a second flowchart illustrating a flow of process executed bythe information processing apparatus of Example Embodiment 2.

FIG. 11 is a block diagram illustrating a functional configuration of aninformation processing apparatus of Example Embodiment 3.

FIG. 12 is a flowchart illustrating a summary of a flow of processexecuted by the information processing apparatus of Example Embodiment3.

FIG. 13 is a diagram illustrating association between a person detectedfrom a video frame at a first time point and a tracking target person.

DESCRIPTION OF EMBODIMENTS

Hereinafter, example embodiments of the present invention will bedescribed using the drawings. It should be noted that in all of thedrawings, the same constituents will be designated by the same referencesigns, and descriptions of such constituents will not be repeated. Inaddition, in each block diagram, unless otherwise particularlydescribed, each block does not represent a hardware unit configurationand represents a function unit configuration.

Example Embodiment 1 Summary

FIG. 1 is a diagram for describing a summary of operation of aninformation processing apparatus (information processing apparatus 2000in FIG. 3 described below) of Example Embodiment 1. The operation of theinformation processing apparatus 2000 in the following description is anillustration for easy understanding of the information processingapparatus 2000. The operation of the information processing apparatus2000 is not limited to the following example. Details and variations ofthe operation of the information processing apparatus 2000 will bedescribed below.

The information processing apparatus 2000 detects a person 20 from videodata 12 that is generated by a camera 10. In FIG. 1 , four persons aredetected from a video frame 14. The video frame 14 is one of capturedimages in time series constituting the video data 12. That is, the videodata 12 is configured with a plurality of video frames 14 in timeseries.

The information processing apparatus 2000 estimates a body height of theperson 20 by performing image analysis on the plurality of video frames14 in which the person 20 is included. A person who is a target forestimating the body height is referred to as a target person. The bodyheight is an invariant quantity that does not change during a certainobservation period (for example, from entrance to exiting of a certaincustomer in a case of trajectory analysis in a store) and is unique toeach person. On the other hand, a height of the person observed from theimage changes due to a change in pose and the like. Therefore, incontrast with the “body height” which is the invariant quantity, theheight of the person observed from the image generated at a certain timepoint will be used distinctively from the body height by referring tothe height of the person as a “temporary-height”. Hereinafter, a meaningof performing image analysis on a part of the video frames 14 will bedescribed.

FIG. 2 is a diagram illustrating the video frame 14 in which the targetperson is included. The person 20 is bending in a video frame 14-1. Onthe other hand, the person 20 is upright in a video frame 14-2.

The body height of the person 20 is the height of the person 20 in anupright state. Thus, even in a case where image analysis is performed onthe video frame 14 such as the video frame 14-1 in which the bendingperson 20 is included, it is difficult to accurately estimate the bodyheight of the person 20. On the other hand, in a case where imageanalysis is performed on the video frame 14 such as the video frame 14-2in which the upright person 20 is included, the body height of theperson 20 can be accurately estimated.

Therefore, the information processing apparatus 2000 estimates a stateof the target person included in the video frame 14 for each of theplurality of video frames 14 in which the target person is included.Furthermore, the information processing apparatus 2000 estimates thebody height of the target person on the basis of the height of thetarget person in the video frame 14 in which the estimated state of thetarget person satisfies a predetermined condition among the plurality ofvideo frames 14 in which the target person is included. It is assumedthat the predetermined condition is a condition that is satisfied in acase where the state of the target person is an upright pose or a poseclose to upright.

For example, in the case in FIG. 2 , the body height of the targetperson is estimated using a height d2 of the target person in the videoframe 14-2 instead of a height d1 of the target person in the videoframe 14-1. The body height that is estimated will be referred to as an“estimated body height”. The estimation of the body height will bereferred to as “computation of the estimated body height”.

Advantageous Effect

As illustrated in FIG. 2 , the target person in a state inappropriatefor the estimation of the body height may be included in the video frame14 in which the target person is included. In a case where the bodyheight of the target person is estimated using such a video frame 14,the body height of the target person cannot be accurately estimated.

Therefore, the information processing apparatus 2000 of the presentexample embodiment estimates the state of the target person included inthe video frame 14 and estimates the body height of the target person onthe basis of the height of the target person in the video frame 14 inwhich the target person in a state appropriate for the estimation of thebody height is included. By doing so, the body height of the targetperson can be accurately estimated.

Hereinafter, the information processing apparatus 2000 of the presentexample embodiment will be described in further detail.

Example of Functional Configuration of Information Processing Apparatus2000

FIG. 3 is a diagram illustrating a configuration of the informationprocessing apparatus 2000 of Example Embodiment 1. For example, theinformation processing apparatus 2000 includes a detection unit 2020, astate estimation unit 2040, and a body height estimation unit 2080. Thedetection unit 2020 detects the person 20 from the video frame 14. Thestate estimation unit 2040 estimates the state of the detected person20. The body height estimation unit 2080 estimates the body height ofthe person 20 on the basis of the height of the person 20 in the videoframe 14 in a case where the estimated state satisfies the predeterminedcondition.

Hardware Configuration of Information Processing Apparatus 2000

Each functional configuration unit of the information processingapparatus 2000 may be implemented by hardware (example: a hardwiredelectronic circuit) implementing each functional configuration unit, ormay be implemented by a combination of hardware and software (example: acombination of an electronic circuit and a program controlling theelectronic circuit). Hereinafter, a case where each functionalconfiguration unit of the information processing apparatus 2000 isimplemented by a combination of hardware and software will be described.

FIG. 4 is a diagram illustrating a computer 1000 for implementing theinformation processing apparatus 2000. The computer 1000 is anycomputer. For example, the computer 1000 is a Personal Computer (PC), aserver machine, a tablet terminal, or a smartphone. Besides, forexample, the computer 1000 may be the camera 10. The computer 1000 maybe a dedicated computer designed to implement the information processingapparatus 2000 or may be a general-purpose computer.

The computer 1000 includes a bus 1020, a processor 1040, a memory 1060,a storage device 1080, an input-output interface 1100, and a networkinterface 1120. The bus 1020 is a data transfer path for transmissionand reception of data among the processor 1040, the memory 1060, thestorage device 1080, the input-output interface 1100, and the networkinterface 1120. A method of connecting the processor 1040 and the liketo each other is not limited to bus connection. The processor 1040corresponds to various processors such as a Central Processing Unit(CPU) and a Graphics Processing Unit (GPU). The memory 1060 is a mainstorage apparatus that is implemented using a Random Access Memory (RAM)or the like. The storage device 1080 is an auxiliary storage apparatusthat is implemented using a hard disk, a Solid State Drive (SSD), amemory card, a Read Only Memory (ROM), or the like. The storage device1080 may be configured by the same hardware as hardware such as the RAMconstituting the main storage apparatus.

The input-output interface 1100 is an interface for connecting thecomputer 1000 to input-output devices. The network interface 1120 is aninterface for connecting the computer 1000 to a communication network.For example, the communication network is a Local Area Network (LAN) ora Wide Area Network (WAN). A method of connecting the network interface1120 to the communication network may be wireless connection or may bewired connection.

For example, the computer 1000 is communicably connected to the camera10 through the network. A method of communicably connecting the computer1000 to the camera 10 is not limited to connection through the network.In addition, the computer 1000 may not be communicably connected to thecamera 10.

The storage device 1080 stores a program module that implements eachfunctional configuration unit (the detection unit 2020, the stateestimation unit 2040, and the height estimation unit 2080) of theinformation processing apparatus 2000. The processor 1040 implements afunction corresponding to each program module by reading each programmodule into the memory 1060 and executing the program module.

It should be noted that the computer 1000 may be implemented using aplurality of computers. For example, each of the detection unit 2020,the state estimation unit 2040, and the height estimation unit 2080 canbe implemented by a different computer. In this case, the program modulestored in the storage device of each computer may be only a programmodule corresponding to the functional configuration unit implemented bythe computer.

Camera 10

The camera 10 is any camera that generates the video data 12 bygenerating the video frame 14 in time series by repeatedly performingcapturing. For example, the camera 10 is a surveillance camera that isinstalled for surveillance of a determined facility, road, and the like.

As described above, the computer 1000 implementing the informationprocessing apparatus 2000 may be the camera 10. In this case, the camera10 performs detection of the person 20, estimation of the state of thetarget person, and estimation of the body height of the target person byanalyzing the video frame 14 generated by the camera 10. As the camera10 having such a function, for example, a camera called an intelligentcamera, a network camera, or an Internet Protocol (IP) camera can beused.

It should be noted that not all functions of the information processingapparatus 2000 may be implemented by the camera 10, and only a part ofthe functions of the information processing apparatus 2000 may beimplemented by the camera 10. For example, only the function ofdetecting the person 20 from the video frame 14 is implemented by thecamera 10, and the other functions of the information processingapparatus 2000 are implemented by a server apparatus. In this case, theserver apparatus acquires various information such as the position andan image feature of the detected person 20 from the camera 10. Inaddition, the server apparatus may acquire only a partial image regionin which the person 20 and its surrounding area are included in thevideo frame 14 generated by the camera 10.

Flow of Process

FIG. 5 is a first flowchart illustrating a flow of process executed bythe information processing apparatus 2000 of Example Embodiment 1. Aloop process A is a loop process that is repeatedly executed until apredetermined end condition is satisfied. The detection unit 2020acquires the most recent video frame 14 (S104). The detection unit 2020detects the person 20 from the acquired video frame 14 (S106). A loopprocess B is a process that is executed for each detected person 20. Theperson 20 of a processing target in the loop process B will be referredto as a person i. The person i is the target person.

FIG. 6 is a second flowchart illustrating a flow of process executed bythe information processing apparatus 2000 of Example Embodiment 1. Inthis case, the detection unit 2020 further associates the detectedperson with a tracking result up to the present, and obtains thetracking result (S120). The loop process B is a process that is executedfor each person 20 included in a tracking target obtained by a trackingprocess. The person 20 of the processing target in the loop process Bwill also be referred to as the person i.

The state estimation unit 2040 estimates the state of the person i (S110and S130). The body height estimation unit 2080 determines whether ornot the state of the person i satisfies the predetermined condition(S112 and S132). In a case where the state of the person i satisfies thepredetermined condition (S112 and S132: YES), the estimated body heightof the person i is computed on the basis of the height of the person iobtained from the video frame 14 acquired in S102 (S114 and S134). In acase where the state of the person i does not satisfy the predeterminedcondition (S112 and S132: NO), the estimated body height of the person iis not computed.

It should be noted that a condition for ending the loop process A is notlimited. For example, the loop process A is ended in a case where apredetermined input operation is received from a user.

Method of Acquiring Video Frame 14: S104

The information processing apparatus 2000 acquires one or more videoframes 14 as a processing target. Various methods of acquiring the videoframe 14 by the information processing apparatus 2000 are present. Forexample, the information processing apparatus 2000 receives the videoframe 14 transmitted from the camera 10. Alternatively, for example, theinformation processing apparatus 2000 accesses the camera 10 andacquires the video frame 14 stored in the camera 10.

It should be noted that the camera 10 may store the video frame 14 in astorage apparatus that is installed outside the camera 10. In this case,the information processing apparatus 2000 acquires the video frame 14 byaccessing the storage apparatus. Thus, in this case, the informationprocessing apparatus 2000 and the camera 10 may not be communicablyconnected.

In a case where a part or all of the functions of the informationprocessing apparatus 2000 are implemented by the camera 10, theinformation processing apparatus 2000 acquires the video frame 14generated by the information processing apparatus 2000. In this case,for example, the video frame 14 is stored in a storage apparatus (forexample, the storage device 1080) inside the information processingapparatus 2000. Therefore, the information processing apparatus 2000acquires the video frame 14 from the storage apparatus.

A timing at which the information processing apparatus 2000 acquires thevideo frame 14 is not limited. For example, each time a new video frame14 constituting the video data 12 is generated by the camera 10, theinformation processing apparatus 2000 acquires the newly generated videoframe 14. Besides, for example, the information processing apparatus2000 may periodically acquire a non-acquired video frame 14. Forexample, in a case where the information processing apparatus 2000acquires the video frame 14 once in one second, the informationprocessing apparatus 2000 collectively acquires a plurality of videoframes 14 (for example, 30 video frames 14 in a case where a frame rateof the video data 12 is 30 frames/second (fps)) that are generated inone second by the camera 10.

The information processing apparatus 2000 may acquire all video frames14 constituting the video data 12 or may acquire only a part of thevideo frames 14. In the latter case, for example, the informationprocessing apparatus 2000 acquires the video frames 14 generated by thecamera 10 at a ratio of one to a predetermined number.

Detection of Person 20: S106

The detection unit 2020 detects the person 20 from the video frame 14(S102). Various known technologies can be used as a technology fordetecting the person 20 from the video frame 14. For example, thedetection unit 2020 includes a detector that learns an image feature ofthe person. The detector detects an image region matching the learnedimage feature from the video frame 14 as a region (hereinafter, a personregion) representing the person 20. For example, a detector thatperforms detection based on a Histograms of Oriented Gradients (HOG)feature or a detector that uses a Convolutional Neural Network (CNN) canbe used as the detector. It should be noted that the detector may be adetector trained to detect the region of the whole person 20 or adetector trained to detect a part of the region of the person 20. Forexample, in a case where a head part position and a foot position can bedetected using the detector that has learned a head part and a foot, theperson region can be determined. Besides, for example, it may beconfigured that the person region is obtained by combining silhouetteinformation (information of a region having a difference with abackground model) obtained by background subtraction with detectioninformation on the head part.

The detector outputs information (hereinafter, detection information)related to the detected person 20. For example, the detectioninformation indicates a position and a size of the person 20. Theposition of the person 20 in the detection information may berepresented as a position on the video frame 14 (for example,coordinates using the upper left end of the video frame 14 as an origin)or may be represented as real world coordinates. Existing technologiescan be used as a technology for computing the real world coordinates ofan object included in an image generated by a camera. For example, thereal world coordinates of the person 20 can be computed from theposition on the video frame 14 using a camera parameter.

For example, the size of the person 20 is represented by a size (forexample, lengths of vertical and horizontal edges or an average valuethereof) of a circumscribed rectangle (hereinafter, referred to as aperson rectangle) of the person or a circumscribed rectangle of a partof the region of the person such as the head part or the foot. This sizemay be a size in the video frame 14 or a size in a real world.

Tracking of Person 20: S120

It should be noted that in a case of the flow illustrated in FIG. 6 ,the detection unit 2020 further performs the tracking process on theperson. The tracking process is a process of associating the persondetected from the video frame input at the current time point with aperson who is detected in the frame in the past and is being tracked.For example, a technology that will be described in Example Embodiment 3described later can be used as a technology for tracking the same personacross the plurality of video frames 14. Other existing technologies canalso be used.

The information processing apparatus 2000 generates tracking informationrepresenting a history of the position and the size of the person 20 bytracking the person 20. A person who is registered in the trackinginformation, that is, a person who is already detected from the videoframe 14 and is set as a tracking target, will be referred to as atracking target person.

FIG. 7 is a diagram illustrating the tracking information. In thetracking information in FIG. 7 , a table 200 is generated for eachtracking target person. The table 200 shows a history of the positionand the like of the associated tracking target person. The table 200shows a frame ID 202, a position 204, a state 206, an observation value208, a motion 210, a feature value 212, and a region 214. The frame ID202 is an identifier of the video frame 14. For example, a record inwhich the frame ID 202 shows n shows the position and the like of thetracking target person in the video frame 14 of which the frame ID is n.It should be noted that the motion 210 shows a parameter of a motionmodel that is used for predicting movement of the tracking target personin a movement state. The motion model will be described in ExampleEmbodiment 3 described later. In addition, TL and BR in the region 214represent the coordinates of the upper left end (top left) and thecoordinates of the lower right end (bottom right), respectively. Theobservation value 208 will be described later.

In the case of the flow illustrated in FIG. 6 , the detection unit 2020may also extract information necessary for tracking at a time ofdetection in S106. That is, the detection information may include afeature value representing an appearance feature of the person 20. Afeature value describing the size of the person 20, a color or a patternof a surface (a color or a pattern of clothes), or the like can be usedas the feature value. More specifically, a feature value such as a colorhistogram, a color layout, an edge histogram, or a Gabor feature can beused. In addition, the detection information may include a feature valuerepresenting a shape of an object. For example, a shape descriptorstandardized in MPEG-7 can be used as the feature value representing theshape of the object. Besides, for example, a keypoint of the person 20may be extracted, and a local feature value such as Scale InvariantFeature Transform (SIFT) or Speeded Up Robust Features (SURF) may beextracted for each keypoint. Besides, for example, feature extractionbased on a network learned by deep learning may be used.

Target Person: S108

The information processing apparatus 2000 sets at least one person 20detected from the video frame 14 as a target (that is, the targetperson) of the process of estimating the body height. In a case where aplurality of persons 20 are detected from the video frame 14, a methodof deciding which person 20 is to be handled as the target person is notlimited. For example, in the process illustrated in the flowchart inFIG. 5 , all persons 20 detected from the video frame 14 are handled asthe target person.

Besides, for example, the information processing apparatus 2000 mayhandle only the person 20 included in a predetermined image region inthe video frame 14 as the target person. The predetermined image regionmay be a predetermined region or a region that is dynamicallydetermined. In the latter case, for example, the information processingapparatus 2000 detects a predetermined object from the video frame 14and handles, as the target person, each person 20 who is detected fromthe image region having a predetermined size using the object as areference. The predetermined object may be an object (a chair, a shelf,a door, a statue, or the like) that is placed at all times, or an objectsuch as a left object that dynamically appears. It should be noted thatexisting technologies can be used as a technology for detecting theobject placed at all times or the dynamically appearing object from theimage.

Estimation of State: S110

The state estimation unit 2040 estimates the state of the target person(S110). The state estimated by the state estimation unit 2040 includesat least information related to a pose of the target person (forexample, information indicating any of upright and non-upright). Variousmethods can be employed as a method of estimating the state of thetarget person. Hereinafter, several specific examples of the method willbe described.

Method 1

First, a case (a case of the process flow in FIG. 5 ) where the bodyheight is computed from only one image of the video frames 14 will bedescribed. In this case, for example, the state estimation unit 2040includes a discriminator (hereinafter, a state discriminator) thatdiscriminates a state by learning, and performs state determinationusing the discriminator. For example, learning data of an imagecorresponding to a state (person pose) of each of “upright” and“non-upright” is prepared. The state discriminator is trained to obtaina correct state when the image of the learning data and a detectionresult of a person are input. In this learning, various discriminatorssuch as a Support Vector Machine (SVM) and a neural network can be used.

More specifically, the state discriminator is trained to distinguish anon-upright state such as crouching or bending from an upright state.

For example, the state estimation unit 2040 inputs the video frame 14and the person detection result with respect to the input video frame 14into the state discriminator. The state discriminator outputsinformation (hereinafter, state information) representing the state ofeach person 20 detected in the video frame 14. Besides, for example, thestate estimation unit 2040 may input only the image region of the targetperson detected from the video frame 14 into the state discriminator ormay input the detection information of the target person into the statediscriminator.

The state information may be information for determining the state ofthe pose of the target person or information indicating a likelihood ofthe target person being in each state. In the latter case, for eachstate which the target person may take, the state discriminator computesa likelihood that the target person is in the state. The statediscriminator includes the likelihood of each state in the stateinformation.

It should be noted that the state estimation unit 2040 may be configuredto determine pose information in which the upright state is furtherclassified in detail as a state. By considering these information,accuracy of the estimation of the height can be increased as will bedescribed later.

For example, in a case of a state of upright and walking, despite of thesame upright state, a pose of walking with legs open and a pose in whichfeet are aligned are considered. Thus, the state discriminator maydistinguish both poses by training the state discriminator todistinguish the poses. Furthermore, in a state where legs are open, thedetermination may be performed by dividing the case into several levelsdepending on a degree of openness. Such a state discriminator alsooutputs information representing whether the legs of the target personare closed or open, information representing the degree of openness ofthe legs, and the like.

In addition, the state information may further include a direction ofthe person. A determination of the direction can also be performed bylearning of the state discriminator in advance.

Method 2

Next, a case (the case of the flow in FIG. 6 ) where the trackingprocess using the plurality of video frames 14 is included will bedescribed. It should be noted that Method 2 is a method of determiningthe state in the state estimation unit 2040 using person rectangleinformation in time series included in the tracking information withoutperforming the pose determination based on the image. A case ofperforming the pose determination based on the image as in Method 1 willbe described later as Method 3.

The state estimation unit 2040 estimates the state of the target personin a certain video frame 14 using a tracking result of the target personin the video frame 14 and across the plurality of video frames 14 in thepast. It should be noted that existing technologies can be used as atechnology for tracking the same person across the plurality of videoframes 14. In addition, the information processing apparatus 2000 mayuse the estimated body height of the person 20 estimated by the bodyheight estimation unit 2080 in tracking of the person 20. A specificmethod will be described in an example embodiment described later.

In a case where the tracking result can be used, it is possible todistinguish a “movement state” and a “standstill state” besides the“upright state” and the “non-upright state”. In this case, for example,a determination of any of three states of a “upright movement state”, a“upright standstill state”, and a “non-upright standstill state” isperformed. It should be noted that while a state referred to as anon-upright movement state may also be included as a selection, it isassumed that the state is not included as a selection in the followingdescription because the state is not usually assumed. Hereinafter,first, distinction between upright and non-upright will be described,and then, distinction between movement and a standstill will bedescribed.

The information processing apparatus 2000 generates the trackinginformation representing the history of the position and the size of theperson 20 by tracking the person 20. The person who is registered in thetracking information, that is, the person who is already detected fromthe video frame 14 and is set as a tracking target, is referred to asthe tracking target person.

While a certain number of histories of the tracking information of thetarget person are stored (while the number of records of the table 200of the target person reaches a certain number), the state estimationunit 2040 computes a height of the target person in the real world fromeach video frame 14. For example, the height of the target person in thereal world is obtained by converting the height of the target person inthe video frame 14 into a real world value using the camera parameter.It should be noted that for example, the height of the target person inthe video frame 14 can be computed as a distance between the foot andthe top head part of the target person in the video frame 14.Hereinafter, the height of the target person in the real world computedfrom the video frame 14 will be referred to as an “observation value”.The observation value is the value shown in the observation value 206,described above. That is, the observation value of the target personcomputed from the video frame 14 is stored in the observation value 206of the tracking information.

In a case where a certain number of observation values are accumulatedfor the target person, the state estimation unit 2040 computes athreshold for identifying whether or not the target person is uprightusing the accumulated observation values. The threshold means a boundaryline between a region in which the person 20 is considered to be uprightand a region in which the person 20 is considered to be not upright in adistribution of the observation values.

After the threshold is computed, the state estimation unit 2040estimates the pose of the target person obtained from the video frame 14using the threshold. For example, in a case where the observation valueof the target person is compared with the threshold and is significantlygreater than the threshold (for example, in a case where the observationvalue is greater than or equal to the threshold), the state estimationunit 2040 estimates that the state of the target person is the uprightstate. On the other hand, in a case where the observation value is notsignificantly greater than the threshold (for example, in a case wherethe observation value is less than the threshold), the state estimationunit 2040 estimates that the state of the target person is thenon-upright state.

Besides, for example, the state estimation unit 2040 may compute anindex value that represents a degree to which the observation valuedeviates from the threshold, and use the index value as a likelihood ofthe state of the target person being the upright state.

A method of deciding the threshold will be described. When a certainnumber of histories are stored, it is considered that the highest valueis close to the body height in a case where there is a significantchange of the height in the real world. Thus, a value obtained bysubtracting a certain value (for example, a value obtained bysubtracting 5 cm) from the height as a reference is set as thethreshold. It should be noted that the subtracted value is preferablydecided by considering error in the observation value. For example,decision can be performed on the basis of a value of a standarddeviation of the error. On the other hand, in a case where there is nosignificant change of the height in the real world (in a case where itis considered that the distribution of the observation values is withina range of the error) when a certain number of histories are stored, avalue obtained by subtracting a certain value from a representativevalue (an average value, a center value, a mode value, or the like) isset as the threshold. In a case where the height in the real worldbecomes greater than the determined threshold with a significantdifference during the setting, it is considered that the original stateis not upright. Thus, the threshold is set again using the newlyobtained observation value as a reference. The state up to the presentis corrected to non-upright and not to upright.

Consideration of Movement and Standstill

As described above, the state which the target person may take mayinclude the upright movement state, the non-upright movement state, andthe upright standstill state. Therefore, it is necessary to determinepresence and absence of movement. Hereinafter, a method of determiningthe presence and the absence of movement and determining any of theupright movement state, the non-upright movement state, and the uprightstandstill state to which the state of the target person correspondswill be described.

The state estimation unit 2040 determines whether or not the targetperson is moving using the tracking information. For example, the stateestimation unit 2040 compares the position of the target person detectedfrom the video frame 14 with the position of the target person apredetermined time before (for example, in the immediately previousframe) and determines that the target person is at a standstill in acase where a change of the position is small (for example, less than orequal to a predetermined value). On the other hand, in a case where thechange of the position is significant, the state estimation unit 2040determines that the target person is moving. The head part position, apredetermined position (for example, a center or an upper left end) inthe person rectangle, the foot position, or the like can be used as theposition of the target person. It should be noted that not only theposition of the target person in the video frame 14 the predeterminedtime before but also the position of the target person in a plurality offrames in the past may be compared.

The state estimation unit 2040 may determine whether or not the targetperson is moving by further considering a direction in which theposition of the target person changes. For example, a change of the headpart of the target person in a vertical direction means that the targetperson stands up or sits down, that is, the pose is changing at the samelocation. Therefore, for example, in a case where the direction in whichthe position of the target person changes is a direction close to thevertical direction, the state estimation unit 2040 determines that thetarget person is not moving. On the other hand, in a case where thedirection in which the position of the head part of the target personchanges is not a direction close to the vertical direction, the stateestimation unit 2040 determines that the target person is moving.

For example, whether or not the direction in which the position of thehead part of the target person changes is close to the verticaldirection can be determined by obtaining an angle between vectors of thedirection in which the position of the target person changes and thevertical direction. It should be noted that in a case where the anglebetween the direction of the change of the position and the verticaldirection is significant, the state estimation unit 2040 may determinethat the target person is moving.

In addition, in a case where a motion of the head part in the image isclose to the vertical direction, the state estimation unit 2040 maydetermine a change in pose and movement by also considering a change ofthe foot position. Specifically, in a case where a change of the footposition is small and is regarded as a standstill, it is considered thatonly the pose is changing and the person is not moving. Thus, the stateof the state estimation unit 2040 is set to the non-upright standstillstate. On the other hand, in a case where the foot position is moving inconjunction with the motion of the head part, the upright movement stateis set.

It should be noted that the foot may be hidden and not seen. Even inthis case, in a case where the foot is regarded as not being inconjunction with the head part (for example, in a case where the headpart is moving in the vertical direction of the image of the camera 10but the foot is hidden by an obstacle or the like), it may be determinedthat the state is the non-upright standstill state by assuming that thefoot position is at a standstill. In addition, in a case of a locationat which a chair is present and sitting is assumed, it is consideredthat a likelihood of sitting is high even in a case where the footposition is not seen. Thus, it may be determined that the state is thenon-upright standstill state. In addition, in a case where acontradiction occurs in a case where a standstill is assumed (forexample, a height of the person rectangle significantly deviates from anassumed height), it may be determined that the state is the uprightmovement state. In a case where it is difficult to determine any of thestates, the likelihoods of both states may be set to be the same, andthe state may be estimated (a significant difference is provided betweenthe likelihoods of the states) at a time point at which any of thestates becomes clear in any frame in the future.

Method 3

Next, a case where the tracking process using the plurality of videoframes 14 is included (the case of the flow in FIG. 6 ) and the posedetermination based on the image is performed in the state estimationunit 2040 will be described. In this case, the pose of the person can bedetermined from the image. Thus, upright and non-upright can be directlyclassified. By combining the pose with the movement informationdescribed in Method 2, three states “upright movement”, “uprightstandstill”, and “non-upright standstill” can be distinguished.

Furthermore, as described in Method 1, a detailed state may be output.That is, information such as the state of walking with the legs open andfurthermore, the degree of openness or the direction of the person maybe output together. This discrimination therebetween can be implementedby generating the discriminator based on the image as described inMethod 1. By considering these information, accuracy of the estimationof the height can be increased as will be described later.

Estimation of Body Height: S114

In the case of the flow in FIG. 5 , when the state of the target personestimated using the video frame 14 satisfies a predetermined condition,the body height estimation unit 2080 estimates the body height of thetarget person on the basis of the height of the target person in thevideo frame 14 (S114). The predetermined condition is a conditionrepresenting that the “state of the target person is the upright stateor a state close to the upright state”.

For example, it is assumed that the state information is information fordetermining the state of the target person. In this case, thepredetermined condition is a condition that the “state of the targetperson is the upright state”.

Besides, for example, it is assumed that the state estimation unit 2040computes the likelihood of each of a plurality of states with respect tothe target person. In this case, the predetermined condition is acondition that the “likelihood of the state of the target person beingthe upright state is greater than or equal to a threshold”.

In a case where the state of the target person in the video frame 14satisfies the predetermined condition, the body height estimation unit2080 computes the observation value of the target person in the videoframe 14. As described above, the observation value of the target personin the video frame 14 is a value obtained by converting the height ofthe target person in the video frame 14 into the height in the realworld using the camera parameter or the like. The detection unit 2020may be configured to compute the observation value of the target personand include the observation value in the detection informationregardless of the state of the target person. In this case, the bodyheight estimation unit 2080 acquires the observation value from thedetection information.

For example, in a case where the state of the target person in the videoframe 14 satisfies the predetermined condition, the body heightestimation unit 2080 sets the observation value of the target person inthe video frame 14 as the estimated body height of the target person.

Besides, for example, in a case where the state estimation unit 2040also outputs more detailed state information (for example, the state ofwalking with the legs open, the degree of openness of the legs, and thedirection of the person), these conditions may also be included in thepredetermined condition. For example, even in the same upright state,the body height may be estimated only in a case where the legs are notopen (alternatively, the degree of openness is small). Alternatively,the body height may be estimated only in a case where the direction ofthe person is significantly different from an optical axis direction ofthe camera.

Estimation of Height: S134

Even in the case of the flow in FIG. 6 , when the state of the targetperson estimated using the video frame 14 satisfies a predeterminedcondition, the body height estimation unit 2080 estimates the bodyheight of the target person on the basis of the height of the targetperson in the video frame 14 (S134). The predetermined condition is acondition representing that the “state of the target person is theupright state or a state close to the upright state”. That is, the stateis the upright standstill state or the upright movement state. Besides,for example, in a case where the state estimation unit 2040 computes thelikelihood of each of the plurality of states with respect to the targetperson, the predetermined condition is a condition that the “likelihoodof the state of the target person being the upright state is greaterthan or equal to a threshold”.

In a case where the state of the target person in the video frame 14satisfies the predetermined condition, the body height estimation unit2080 computes the observation value (the height in the real world) ofthe target person in the video frame 14. The detection unit 2020 may beconfigured to compute the observation value of the target person andinclude the observation value in the detection information regardless ofthe state of the target person. In this case, the body height estimationunit 2080 acquires the observation value from the detection information.

For example, the body height estimation unit 2080 computes the estimatedbody height of the target person using not only the observation valueobtained in the most recent video frame 14 but also the observationvalue obtained from the video frame 14 (one or plural) in the past inwhich the state of the target person satisfies the predeterminedcondition. Specifically, a statistic value of the observation valueobtained from the most recent video frame 14 and the observation valueobtained from the video frame 14 in the past is computed, and thestatistic value is set as the estimated body height of the targetperson.

For example, in a case where the state of the target person in the mostrecent video frame 14 satisfies the predetermined condition, the bodyheight estimation unit 2080 computes the estimated body height of thetarget person using the observation value obtained from the video frame14 and the observation value obtained from each of a predeterminednumber of video frames 14 in the part in which the state of the targetperson satisfies the predetermined condition. By computing the estimatedbody height of the target person each time the video frame 14 in whichthe state of the target person satisfies the predetermined condition isobtained, the estimated body height of the target person can begradually updated to a more accurate value.

Various statistic values can be used. For example, a statistic processis an average value. However, the estimated body height may not becorrectly obtained as in a case of erroneous detection. Thus, thestatistic value may be computed by a statistic process of excluding anoutlier using a method such as robust estimation. For example, a methodsuch as Random Sampling Consensus (RANSAC) can be used.

Besides, for example, the observation values may be weighted and addeddepending on the likelihood of the state of the target person. Forexample, in a case where an angle of depression is increased due to thetarget person positioned close to the camera, it is difficult toestimate the top head part of the target person, and the observationvalue of the target person is likely to include error. On the otherhand, in a case where the target person is distant from the camera,resolution (the size in the video frame 14) of the target person isdecreased, and a slight deviation of a detection position affects theestimation of the body height. Thus, depending on the angle ofdepression from the camera and the distance from the camera, theobservation values may be weighted and added by considering an erroroccurrence likelihood of the observation value, and the average may becomputed. Since a tendency of error corresponding to the angle ofdepression and the distance of the camera can be grasped by capturing aperson having a known height in advance by the camera, the erroroccurrence likelihood can be determined in advance on the basis of aresult of the tendency of error. That is, a weight may be decided suchthat as the error is increased, the weight is decreased, and may be usedin weighted averaging.

In addition, weighting in which a type of state of the person isconsidered may be performed. According to the above stateclassification, the pose at a time of movement is only upright, and thepose at a time of a standstill includes two poses of upright andnon-upright. Thus, in a case of the movement state, there is a highlikelihood that the observation value of the target person representsthe body height. On the other hand, the height is slightly increased ordecreased at the time of movement. Thus, there is a high likelihood thatthe observation value of the target person includes certain error. Inaddition, at the time of the standstill, there is a low likelihood thatthe observation value includes error. Thus, the body height of thetarget person can be estimated with the highest certainty at a time ofthe upright standstill state.

Therefore, a high weight is applied to the observation value of thetarget person obtained from the video frame 14 in which the state of thetarget person is the upright standstill state. For example, immediatelyafter a transition from the movement state to the standstill state, in acase where a state of the measured observation value is stable and isclose to the estimated body height computed at a time of the movementstate, it is regarded that a certainty of a standstill in an uprightpose is high. Thus, estimation may be performed by increasing the weightof the observation value measured in such a situation. In addition, evenin the same movement state, easiness of obtaining also changes dependingon a movement velocity. For example, in a case of fast movement, achange in height of the person is increased along with an increase instep length. Thus, in a case where the velocity is high, control may beperformed to decrease the weight.

Furthermore, the weighting may be performed by considering the directionof the target person, the degree of openness of the legs, and the like.In a case where the target person is in the upright movement state, adegree to which the observation value of the target person representsthe body height varies depending on the direction of movement and thepose of the person at that time. Specifically, in a state where the legsare open, the height is decreased compared to the height in a case wherethe feet are aligned. Thus, in a case where the legs of the targetperson are open, the observation value of the target person at that timeis likely to be a value smaller than the actual body height. Therefore,the weight applied to the observation value of the target person in apose in which the feet are aligned is set to be greater than the weightapplied to the observation value of the target person in a pose in whichthe legs are open.

In addition, in a case where the direction of movement is close to theoptical axis direction of the camera, the rectangular foot positionobtained as the person region is unlikely to be determined, and error islikely to occur in the observation value of the target person. This ismore noticeable in a state where the legs are open. Thus, the statisticprocess is performed by decreasing the weight of the observation valueof the target person moving in a direction close to the optical axisdirection of the camera.

In addition, an approximate value of the body height may be computed ata time of the upright movement state, and a detailed value may beestimated from the approximate value at a time point of a transition tothe standstill state. That is, after the person transitions to theupright standstill state (or before the transition), in a case where theobservation value is stable at a value close to the approximate valuecomputed in the upright movement state, there is a high likelihood ofcomputing a correct body height. Thus, the body height may be computedby applying a high weight to the observation value in such a state. Forexample, the approximate value of the body height is a temporary-heightobtained for the target person in the upright movement state.

It should be noted that in order to accurately compute the estimatedbody height, both of the foot and the head part of the target person arepreferably seen in the video frame 14. However, the foot may be hiddendue to an overlap with the obstacle or another person. A case where thefoot is hidden is desirably excluded in the estimation of the bodyheight. Therefore, a condition that the “foot of the target person isseen” may be added to the predetermined condition. It should be notedthat various methods can be used as a method of determining whether ornot the foot of the target person is seen. Specifically, the foot may bedirectly detected, or a determination as to whether or not the foot ishidden by the obstacle or overlaps with a person may be performed. Theobstacle such as a shelf is usually present at a predetermined position.Thus, a region in which the foot position is hidden on a floor when seenfrom the camera can be determined in advance. Thus, a determination asto whether or not the foot of the target person is seen can be performedby determining whether or not the position of the target person is inthe region. In addition, the overlap with a person can be determined onthe basis of the position of each person 20 detected by the detectionunit 2020. For example, in a case where the circumscribed rectangle ofanother person 20 overlaps with a lower edge of the circumscribedrectangle of the target person, it is determined that the foot of thetarget person is not seen. At this point, a determination as to whetheror not the foot of the target person is seen may be performed using afront-rear relationship between the target person and the other person20 seen from the camera by considering a three-dimensional position ofeach person 20. That is, in a case where the circumscribed rectangles ofthe target person and the other person 20 overlap and the other person20 is closer to the camera than the target person is, it is determinedthat the foot of the target person is not seen.

Example Embodiment 2

FIG. 8 is a diagram illustrating a functional configuration of theinformation processing apparatus 2000 of Example Embodiment 2. Theinformation processing apparatus 2000 of Example Embodiment 2 has thesame function as the information processing apparatus 2000 of ExampleEmbodiment 1 except for the matter described below.

The information processing apparatus 2000 of Example Embodiment 2includes a temporary-height estimation unit 2100. The temporary-heightestimation unit 2100 computes the temporary-height of the target personat a certain time point. The temporary-height is a height that isobtained regardless of whether or not the target person is upright. Forexample, in a case where the target person is crouching, a height fromthe foot to the top head part of the target person in that state is thetemporary-height of the target person at that time. At a time of theupright state, the temporary-height matches the body height in anobservation error range.

Flow of Process

FIG. 9 is a first flowchart illustrating a flow of process executed bythe information processing apparatus 2000 of Example Embodiment 2. Theflowchart in FIG. 9 is different from the flowchart in FIG. 5 in that aprocess (S202) of computing the temporary-height of the target person isexecuted after the estimation of the body height (S114). Other parts arethe same as the flowchart in FIG. 5 .

FIG. 10 is a second flowchart illustrating a flow of process executed bythe information processing apparatus 2000 of Example Embodiment 2. Theflowchart in FIG. 10 is different from the flowchart in FIG. 6 in that aprocess (S212) of computing the temporary-height of the target person isexecuted after the estimation of the body height (S134). Other parts arethe same as the flowchart in FIG. 6 .

Estimation of Temporary-Height: S202

In a case of the flow in FIG. 9 , for example, the temporary-heightestimation unit 2100 sets the observation value of the target person inall frames including the video frame 14 in which the state of the targetperson does not satisfy the predetermined condition, to be thetemporary-height of the target person at a time point at which the videoframe 14 is generated.

Estimation of Temporary-Height: S212

In a case where the tracking process is included (in a case of the flowin FIG. 10 ), for example, the temporary-height estimation unit 2100 maycompute the temporary-height of the target person using also informationobtained from the plurality of video frames 14 in the past. Thetemporary-height of the target person has characteristics of aconsecutive change in time and characteristics of being less than orequal to the estimated body height. Thus, as in a case where the foot ofthe target person is not seen due to the obstacle or the like, in a casewhere it is considered that reliability of the observation valuecomputed from one video frame 14 is low, the temporary-height of thetarget person is preferably computed by performing complementation(interpolation and extrapolation) in a time direction in considerationof consecutiveness. It should be noted that any existing methods can beused as a method of complementation.

For example, it is assumed that the video frame 14 generated at timepoint t does not satisfy the predetermined condition for computing theestimated body height and the foot of the target person is not seen inthe video frame 14. In this case, for example, the temporary-heightestimation unit 2100 computes the temporary-height of the target personat time point t by performing complementation on a change in time of thetemporary-height of the target person computed from each of theplurality of video frames 14 in the past before time point t.

It should be noted that one temporary-height of the target person may becomputed using a plurality of video frames 14 adjacent in time. Forexample, the observation values of the target person computed for videoframes 14 are averaged in a certain width of a time window. By doing so,the temporary-height of the target person is obtained for each timewindow. At this point, a weight corresponding to the certainty of theobservation value may be applied to the observation value obtained fromeach video frame 14.

In addition, when the change in pose is small and the observation valueof the target person is stable, the width of the time window may beincreased. When the height rapidly changes in one direction due tositting down or standing up, the width of the time window may bedecreased.

In a specific state of the person, it is desirable to perform estimationbased on constraints described below. In the non-upright standstillstate such as crouching, bending, and sitting, it is considered that theposition of the person does not change. This tendency is particularlystrong in an area in which particularly, a chair or the like is presentand sitting is assumed. Thus, in the non-upright standstill state, evenin a case where the foot is not seen, the temporary-height may becomputed by changing only the top head position without moving the footposition. That is, the foot position is set to a common position amongthe plurality of video frames 14.

Example of Hardware Configuration

For example, a hardware configuration of a computer that implements theinformation processing apparatus 2000 of Example Embodiment 2 isrepresented by FIG. 4 in the same manner as Example Embodiment 1.However, the storage device 1080 of the computer 1000 implementing theinformation processing apparatus 2000 of the present example embodimentfurther stores a program module that implements the function of theinformation processing apparatus 2000 of the present example embodiment.

Advantageous Effect

According to the information processing apparatus 2000 of the presentexample embodiment, the estimated body height of the person 20 and thetemporary-height of the person 20 (a height in a case where the person20 is not upright) are distinctively computed on the basis of the stateof the person 20. Thus, the estimated body height of the person 20 canbe accurately computed, and the temporary-height of the person 20 ateach time can also be accurately computed.

Example Embodiment 3

FIG. 11 is a block diagram illustrating a functional configuration ofthe information processing apparatus 2000 of Example Embodiment 3. Theinformation processing apparatus 2000 includes a position estimationunit 2120 and an update unit 2140. The information processing apparatus2000 of Example Embodiment 3 has the same function as the informationprocessing apparatus 2000 of Example Embodiment 2 except for the matterdescribed below.

The information processing apparatus 2000 of Example Embodiment 3 has afunction of tracking the person included in the video data 12.Specifically, the person is tracked by generating and updating thetracking information.

First, an overall flow of person tracking process will be describedusing a flowchart. FIG. 12 is a flowchart illustrating a summary of aflow of process executed by the information processing apparatus 2000 ofExample Embodiment 3. The information processing apparatus 2000 detectsthe person 20 from the most recent video frame 14 (S302). Theinformation processing apparatus 2000 generates the tracking informationusing the detected person 20 as the tracking target person (S304). Thetracking target person means the person 20 who is already detected bythe information processing apparatus 2000.

A loop process C is a process that is repeatedly executed until apredetermined end condition is satisfied. In S306, in a case where thepredetermined end condition is satisfied, the information processingapparatus 2000 ends the process in FIG. 12 . On the other hand, in acase where the predetermined end condition is not satisfied, the processin FIG. 12 proceeds to S308.

In S308, the detection unit 2020 detects the person 20 from the mostrecent video frame 14. A time point at which the acquired video frame 14is generated will be referred to as a first time point. The positionestimation unit 2120 estimates a position of each tracking target personat the first time point using the tracking information (S310). Theupdate unit 2140 associates the person 20 detected from the video frame14 with the tracking target person (S312). The update unit 2140 updatesthe tracking information on the basis of a result of association (S314).

It should be noted that a timing at which the information processingapparatus 2000 of Example Embodiment 3 computes the estimated bodyheight or the temporary-height of the target person is not limited. Forexample, the information processing apparatus 2000 computes theestimated body height or the temporary-height of the target personduring the loop process C.

Computation of Estimated Position: S310

The position estimation unit 2120 estimates a position of each trackingtarget person at the first time point using the tracking information(S310). The position of the tracking target person shown in the trackinginformation is a position in the past (for example, a position in theimmediately previous video frame 14). Therefore, the position estimationunit 2120 estimates the position of the tracking target person at thefirst time point from the position of the tracking target person in thepast.

Various methods are present for the estimation. For example, theposition estimation unit 2120 predicts the position of the trackingtarget person at the first time point on the basis of the position ofthe tracking target person shown in the tracking information and themotion model of the person. As this method, various existing methodssuch as a method of using a Kalman filter and a method of using aparticle filter can be used. It should be noted that predictedpositional information may be coordinates in the image or real worldcoordinates. In a case of the coordinates in the image, a predictionresult may be generated as the person rectangle information.

The motion model of the person may vary for each tracking target person.For example, the position estimation unit 2120 decides the motion modelof the tracking target person using a state that is estimated in thepast for the tracking target person. For example, a prediction modelcorresponding to each of the upright movement state, the uprightstandstill state, and the non-upright standstill state is prepared inadvance. The position estimation unit 2120 computes an estimatedposition of the tracking target person using the prediction modelcorresponding to the most recent state of the tracking target person(the state of the tracking target person shown in the most recenttracking information).

For example, in the upright movement state, a model that predicts amotion based on the history of the position of the tracking targetperson shown in the tracking information is used. As a specific example,the following uniform linear motion model can be used. First, theuniform linear motion model computes a velocity vector of the trackingtarget person using the tracking information in a predetermined periodin the past and computes a motion vector of the tracking target persondepending on the velocity vector and a frame time interval. The uniformlinear motion model computes the estimated position of the trackingtarget person at the first time point by adding the motion vector to theposition of the tracking target person shown in the most recent trackinginformation.

In the upright standstill state, it can be predicted that the trackingtarget person does not move. Thus, a model having a movement amount of 0is used. That is, the position of the tracking target person at thefirst time point is the same as the position shown in the most recenttracking information.

In the non-upright standstill state, movement caused by walking does notoccur, but a change in pose may occur. For example, while the footposition does not move, the head part position may move in a directionclose to verticality. Therefore, in the non-upright standstill state, amodel that predicts such a change in pose is used.

The position estimation unit 2120 may estimate the position byconsidering the state of the tracking target person only in a statehaving the highest certainty (most likely state) or in each of aplurality of states. In the latter case, the estimated positioncorresponding to each of the plurality of states is computed.

Association between Person 20 and Tracking Target Person: S312

The update unit 2140 associates the person 20 detected from the videoframe 14 at the first time point with the tracking target person. Thisassociation is a process of determining the tracking target person towhich each person 20 detected from the video frame 14 at the first timepoint corresponds. FIG. 13 is a diagram illustrating association betweenthe person 20 detected from the video frame 14 at the first time pointand the tracking target person. In FIG. 13 , the person 20 and thetracking target person who are connected to each other by abidirectional arrow are the person 20 and the tracking target person whoare associated with each other.

Various methods are present for the association. For example, on thebasis of closeness between the predicted position of the tracking targetperson and the position of the detected person 20, similarity betweenappearance feature values of the both, and the like, the body heightestimation unit 2080 can compute a likelihood (hereinafter, referred toas an association likelihood) representing a possibility of associationbetween the both and perform the association. For example, thelikelihood can be converted into a cost, and the association problembetween the tracking target person and the detected person can bereduced to a minimum cost problem and be solved using an algorithm suchas a Hungarian method.

In a case where the association likelihood is obtained, for example,closeness information of the person is determined by a distance in thereal space. That is, positions in the video frame 14 are converted intoposition coordinates in the real space using the camera parameter andheight information of the persons, and a distance between thecoordinates is obtained.

At this point, the update unit 2140 decides the height of the person tobe used in the conversion, depending on the state of the person. Asdescribed above, the information processing apparatus 2000 distinctivelycomputes the estimated body height and the temporary-height of thetarget person. In addition, the temporary-height of the target person ateach time point is stored in the tracking information.

Therefore, the update unit 2140 computes the position using any of theestimated body height and the temporary-height of the person 20depending on the state of the person 20 at the first time point.Specifically, in the upright standstill state or the upright movementstate, either the person 20 is at a standstill or is moving in a uprightstate. Thus, the height of the person 20 may be regarded as being equalto the body height. Thus, the update unit 2140 regards the height of thetop head position of the person 20 detected from the video frame 14 asthe body height and converts the coordinates of the person 20 in thevideo frame 14 into real world coordinates using the camera parameter.On the other hand, in a case of the non-upright standstill state, theheight of the person 20 is different from the height and thus, iscomputed using the temporary-height of the person 20 at the first timepoint.

It should be noted that as described above, in a case where a motion ofthe head part of the person 20 is also predicted in the case of thenon-upright standstill state in the position estimation unit 2120, theposition of the person 20 may be computed by reflecting a change of thetemporary-height of the person 20 caused by the motion. That is, withoutusing the temporary-height of the person 20 included in the trackinginformation in the past as it is, the positional information may beestimated by reflecting the change caused by the motion of the head parton the temporary-height of the person 20.

In addition, the closeness information of the person may be computed ascoordinates in the image. For example, coordinates of a specificposition (for example, the foot position) of the person rectangle in theimage may be obtained, and the closeness may be determined by thedistance between the coordinates. Alternatively, the closeness may bedetermined by an overlap between the person rectangles. In this case, itis determined that as the overlap between the rectangles is increased,the persons are closer to each other.

It should be noted that the estimated position of the tracking targetperson is assumed to be obtained in only the most likely state. In thiscase, the update unit 2140 may obtain the association likelihood betweenthe position of each tracking target person in the most likely state andthe position of the person 20 detected from the video frame 14 at thefirst time point. On the other hand, in a case where the estimatedposition of the tracking target person is obtained for the plurality ofstates, the association likelihood is computed for each of the states,and the state having the highest association likelihood and theassociation likelihood at that time are selected.

In addition, the similarity between the appearance feature values may beconsidered together. In this case, a determination is performed using adistance between the feature values or a scale representing thesimilarity. As the scale, various existing scales such as a Euclideandistance and a histogram intersection can be used.

Update of Tracking Information

The update unit 2140 updates information of the tracking target personshown in the tracking information on the basis of the result ofassociation. Specifically, information related to each tracking targetperson at the first time point is added to the tracking information.

For example, the update unit 2140 updates the positional information ofthe tracking target person. For example, the update unit 2140 sets theposition of the tracking target person at the first time point as theposition of the person 20 associated with the tracking target person.Besides, for example, the update unit 2140 may set the positionalinformation of the tracking target person at the first time point as aposition obtained by weighting and adding the estimated position of thetracking target person at the first time point and the position of theperson 20 associated with the tracking target person.

Regarding motion information of the tracking target person, a parameterof the motion model is updated on the basis of a difference between theupdated positional information and prediction information of the motion.For example, in a case of a person who can be assumed to have a uniformlinear motion, it is considered that an update is performed by addingthe difference between the estimated position and the updated positionto the current motion. Alternatively, in a case where tracking isperformed using the Kalman filter, the positional information and themotion information may be updated on the basis of a known update formulaof the Kalman filter.

Regarding an update of region information, the rectangle may be computedagain on the basis of the updated positional information in a case wherethe region information is the circumscribed rectangle of the person. Atthis point, the rectangle information may be updated by considering notonly movement of the position but also a change of size in appearanceand the like using calibration information of the camera.

The corresponding tracking target person may not be present for theperson 20 detected from the video frame 14. For this person 20, adetermination as to whether or not the person 20 is a newly appearingperson (enters a capturing range of the camera 10) is performed. In acase where it is determined that the person 20 is a newly appearingperson, the person 20 is added to the tracking information as a newtracking target person. On the other hand, in a case where the person 20is not a newly appearing person, the person 20 is not added to thetracking information. For example, in a case where the region of theperson 20 significantly overlaps with the region of the existingtracking target person, it is determined that there is a highpossibility of erroneous detection, and new addition is not performed.

On the other hand, the corresponding person 20 may not be present forthe tracking target person. For this tracking target person, adetermination as to whether or not the tracking target person is aperson out of the capturing range of the camera 10 is performed. In acase where it is determined that the tracking target person is a personout of the capturing range, the person is excluded from the trackinginformation. For example, in a case where the tracking target personmoving toward the outside of the capturing range around an edge of thecapturing range of the camera 10 at a time of a previous update or thetracking target person present near an exit at a time of the previousupdate is not associated with any person 20, the tracking target personis a person out of the capturing range of the camera 10. Instead ofexcluding the person from the tracking information, information thatindicates that the person out of the capturing range of the camera 10may be added to the tracking information (for example, a bit of anexclusion flag is set to 1).

Example of Hardware Configuration

For example, a hardware configuration of a computer that implements theinformation processing apparatus 2000 of Example Embodiment 3 isrepresented by FIG. 4 in the same manner as Example Embodiment 1.However, the storage device 1080 of the computer 1000 implementing theinformation processing apparatus 2000 of the present example embodimentfurther stores a program module that implements the function of theinformation processing apparatus 2000 of the present example embodiment.

Advantageous Effect

According to the information processing apparatus 2000 of the presentexample embodiment, in the computation of the estimated position of thetracking target person and the association between the tracking targetperson and the person 20 detected from the video frame 14, the state ofthe person is considered, and any of the estimated body height and thetemporary-height of the person that is more appropriate is used. Bydoing so, the person can be tracked with higher accuracy.

By tracking the person with higher accuracy, the position of the personat each time can be estimated with high accuracy. Accordingly, thetrajectory analysis can be performed with higher accuracy than thetrajectory analysis in the related art. For example, in the store, atrajectory of a customer can be analyzed and used for marketing, or amotion of staff can be visualized and used for a purpose of measuringwork efficiency. In addition, in a warehouse or a factory, how a workeris moving can be visualized and used for reviewing improvement inworkflow and work efficiency.

Example Embodiment 4

For example, the information processing apparatus 2000 of ExampleEmbodiment 4 is shown in FIG. 11 in the same manner as the informationprocessing apparatus 2000 of Example Embodiment 3. The informationprocessing apparatus 2000 of Example Embodiment 4 has the same functionas the information processing apparatus 2000 of Example Embodiment 3except for the matter described below.

In the information processing apparatus 2000 of Example Embodiment 4, itis assumed that the video data 12 is obtained from a plurality ofcameras 10. Thus, the following feature is included.

Generation of Detection Information

The detection unit 2020 generates the detection information such thatthe camera 10 that captures each detected person 20 can be determined.Specifically, an identifier (camera identifier) that indicates thecamera 10 by which information detected from the video frame 14 isgenerated is set in the detection information. For example, thedetection unit 2020 generates individual detection information for eachof the plurality of cameras 10 and associates the camera identifier witheach detection information. Besides, for example, the detection unit2020 may generate one detection information that shows all persons 20detected from the plurality of cameras 10, and indicate the camera 10from which the person is detected in each record.

Update of Tracking Information

In a case where the video frame 14 is obtained from each of theplurality of cameras 10, the detected person 20 varies for each videoframe 14. However, in a case where the capturing ranges of the cameras10 partially overlap, the same person 20 may be detected from theplurality of video frames 14.

Considering that the detected person 20 varies for each video frame 14,the information processing apparatus 2000 narrows down the trackingtarget persons of an update target to a part of the tracking targetpersons included in the tracking information when the trackinginformation is updated using the video frame 14 obtained from a certaincamera 10. For example, it is assumed that the information processingapparatus 2000 updates the tracking information using the video frame 14generated by a first camera. In this case, the position estimation unit2120 computes the estimated position for only the tracking target personhaving a high likelihood of being present in the capturing range of thefirst camera as a target among the tracking target persons shown in thetracking information. For example, the position estimation unit 2120obtains, in advance, a region that can be captured from the first camerain the real space, and extracts the tracking target person who isestimated to be included in the capturing range of the first camera bydetermining whether or not each tracking target person is included inthe region from the positional information. The position estimation unit2120 computes the estimated position for only the extracted trackingtarget person.

In addition, the update unit 2140 performs the association with theperson 20 detected from the video frame 14 and the update of thetracking information for only the extracted tracking target person. Atthis point, for a person who is not included in the capturing range ofthe first camera among the tracking target persons, information thatrepresents that the tracking information is not updated because theperson is not included in the capturing range of the first camera may beincluded in the tracking information, and the person may be used in asubsequent stage of the process.

Estimation of State

In a case where the plurality of cameras 10 are present, easiness ofstate determination of the person may vary depending on the camera. Thestate estimation unit 2040 of Example Embodiment 4 estimates the stateof the person by considering this point. For example, in a case wherethe person is closer to the camera, the size (resolution) of the personregion in the video frame 14 is increased, and the motion and the likeof the person are easily determined. In addition, in a case where themotion of the person is determined, a motion in a directionperpendicular to an optical axis is more easily determined than a motionin the optical axis direction of the camera. Accordingly, the easinessand certainty of state determination of the person change depending on apositional relationship between the camera and the person, and the like.

Therefore, the state estimation unit 2040 estimates the state of eachperson 20 using the video frame 14 obtained from each of the pluralityof cameras 10 and uses a state having the highest reliability among thestates. For example, for each video frame 14, the state estimation unit2040 computes reliability of state estimation on the basis of a distancebetween the camera 10 generating the video frame 14 and the detectedperson 20 (a shorter distance means higher reliability) or arelationship of an angle between the motion of the person 20 and theoptical axis of the camera 10 (an angle closer to perpendicularity meanshigher reliability). The state of the person 20 that is estimated usingthe video frame 14 having the highest reliability is used as the stateof the person 20.

In addition, for each video frame 14, a determination as to whether ornot the target person is hidden by another person or the obstacle may beperformed and be reflected on a priority of the camera for performingstate determination. That is, the state of the target person isestimated by preferentially using the video frame 14 in which the targetperson is not hidden by another person or the obstacle.

Correction of Estimated Body Height and Temporary-Height

In a case where the target person is captured by the plurality ofcameras 10, the body height estimation unit 2080 increases accuracy ofthe estimated body height using the video frame 14 generated by each ofthe plurality of cameras 10 at the same time (that is, using a pluralityof video frames 14 generated by different cameras 10). In a case wherethe estimated body height of the person is different from the actualbody height and the position of the person in the real space isestimated using the estimated body height, the position of the person isprojected to a position that deviates in a depth direction of thecamera. For example, in a case where the estimated body height is lessthan the actual body height and the coordinates in the image areconverted into the coordinates in the real space using the cameraparameter, the converted coordinates are farther from the camera thanthe actual position is. Conversely, in a case where the estimated bodyheight is greater, the converted coordinates are closer than the actualposition is. Thus, in a case where there is a deviation between theestimated body height and the actual body height, even the same personis projected to a different position for each camera in a case where theposition is obtained for each of the plurality of cameras at the sametime. In other words, in a case where the position deviates, theestimated body height can be approximated to a correct value bycorrecting the estimated body height to match the positions.

Specifically, if the difference between the positions in the real spaceobtained as to the cameras decreases in a case where a certain value isadded to the estimated body height comparing to a case of the originalestimated body height, the estimated body height is increased.Conversely, if the difference between the positions in the real spaceobtained as to the cameras decreases in a case where a certain value issubtracted from the estimated body height comparing to a case of theoriginal estimated body height, the estimated body height is decreased.Consequently, in a case where the positions are sufficiently close amongthe cameras, it is considered that the estimated body height at thattime is close to the true value. Thus, the obtained value is set as theestimated body height. In a case where the accuracy of the estimatedbody height at each time point is increased, the accuracy of theestimated body height that is finally computed as the statistic value ofthe estimated body height at each time point is also increased.

It should be noted that the temporary-height of the person can also becorrected using the same method.

Erroneous detection or non-detection may also be included in a result ofperson detection. Thus, a method of gradually changing and setting theestimated body height and the temporary-height along with an elapse oftime without significantly changing the estimated body height and thetemporary-height at once is also considered. In this case, since theestimated body height and the temporary-height gradually change, smoothtrajectory information can be obtained. In addition, high accuracyposition estimation can be performed without being significantlyaffected by a sudden erroneous detection result.

It should be noted that in a case where the person who can be perceivedfrom the plurality of cameras overlaps with another person when seenfrom a certain camera and the person rectangle cannot be estimated withhigh reliability, information of the camera is not used in thecomparison. Whether or not there is an overlap between the trackingtarget persons can be determined by converting the positionalinformation of the tracking target person into a position in the imageusing the camera parameter, estimating a region in which each trackingtarget person is present in the image, and determining whether or notthe region overlaps with the region of another person.

Example of Hardware Configuration

For example, a hardware configuration of a computer that implements theinformation processing apparatus 2000 of Example Embodiment 4 isrepresented by FIG. 4 in the same manner as Example Embodiment 1.However, the storage device 1080 of the computer 1000 implementing theinformation processing apparatus 2000 of the present example embodimentfurther stores a program module that implements the function of theinformation processing apparatus 2000 of the present example embodiment.

Advantageous Effect

According to the information processing apparatus 2000 of the presentexample embodiment, the update of the tracking information, thecomputation of the estimated body height, and the like are performedusing the video frame 14 obtained from the plurality of cameras 10.Thus, the tracking of the person, the estimation of the body height, andthe like can be performed with higher accuracy.

While example embodiments of the present invention have been describedthus far with reference to the drawings, the example embodiments areillustrations of the present invention, and a configuration ofcombinations of the example embodiments or other various configurationscan also be employed.

1. An information processing system comprising: a processor; and memorystoring executable instructions that, when executed by the processor,causes the processor to perform as: a detection unit that detects aperson from a video frame; a state estimation unit that estimates astate of a target person using a result of the detection; and a bodyheight estimation unit that estimates a body height of the target personon the basis of a height of the target person in the video frame in acase where the state of the target person satisfies a predeterminedcondition, wherein the body height estimation unit computes an estimatedbody height of the target person by computing a height of the targetperson in a real world from each of a plurality of video frames in whichthe estimated state satisfies the predetermined condition, andperforming a statistic process on the plurality of computed heights,wherein the state estimation unit computes the height of the targetperson in the real world from each of a plurality of video frames inwhich the target person is included, and decides a threshold fordetermining whether or not the person is upright on the basis of thecomputed height, and for the video frame that is subsequently generated,determines whether or not the target person is upright in the videoframe by computing the height of the target person in the real world andcomparing the height with the threshold, wherein the predeterminedcondition is satisfied in a case where the target person is upright, orthe predetermined condition is that the target person is in apredetermined pose, that likelihood of the target person being in thepredetermined pose is greater than or equal to a threshold, that adirection of the target person is in a predetermined state, or that footof the target person is visible.
 2. The information processing systemaccording to claim 1, wherein the state which the target person may takeincludes an upright movement state, an upright standstill state, and anon-upright standstill state, and the predetermined condition issatisfied in a case where the state of the target person is the uprightmovement state or the upright standstill state.
 3. The informationprocessing system according to claim 1, wherein the body heightestimation unit performs the statistic process on the height of thetarget person in the real world computed from each of the plurality ofvideo frames by applying a weight based on an angle of depression, whichis determined by the target person and a direction of the cameragenerating the video frame and resolution of the target person in thevideo frame.
 4. The information processing system according to claim 1,wherein the state which the target person may take includes an uprightmovement state, an upright standstill state, and a non-uprightstandstill state, and the state estimation unit further estimates atleast one of a degree of openness of legs of the target person or adirection of the target person, and in the statistic process, in a casewhere the state of the target person is the upright movement state, ahigher weight is applied to the height of the target person in the realworld as the degree of openness of the legs of the target person isdecreased, or a higher weight is applied to the height of the targetperson in the real world as a degree to which the direction of thetarget person is different from a direction of the camera capturing thevideo frame is increased.
 5. The information processing system accordingto claim 1, wherein the state which the target person may take includesan upright movement state, an upright standstill state, and anon-upright standstill state, and the body height estimation unitestimates the body height by performing weighting in consideration ofthe state of the target person, computes an approximate value of thebody height using a time of the upright movement state as a reference,and estimates the body height by increasing a weight of an observationvalue that is stable at a value close to the approximate value of thebody height before and after the upright movement state.
 6. Theinformation processing system according to claim 1, further comprising:a temporary-height estimation unit that estimates a temporary-height ofthe target person on the basis of the height of the target person in thevideo frame.
 7. The information processing system according to claim 6,wherein the state which the target person may take includes an uprightmovement state, an upright standstill state, and a non-uprightstandstill state, and even in a case where a foot position of the personis not seen, the temporary-height estimation unit computes anobservation value of the temporary-height even from only a head partposition in the upright standstill state and the non-upright standstillstate by assuming that the foot position in the image does not move. 8.The information processing system according to claim 6, wherein thedetection unit generates a person detection result for each camera byperforming a person detection process on each of the images generated bytwo or more cameras, and for a tracking target person who is present ina region observable by a plurality of cameras, the temporary-heightestimation unit and the body height estimation unit compare positioncoordinates that are included in a tracking result based on a detectionresult of the image that is acquired from each camera with a generationtime difference less than a predetermined time, and correct observationvalues of the body height and the temporary-height such that a distancebetween the coordinates is decreased.
 9. The information processingsystem according to claim 6, wherein the temporary-height estimationunit and the body height estimation unit set the predetermined time tobe short to an extent that the person can be regarded as being presentat the same position when the person is in an upright movement state,and set the predetermined time to be long to an extent that the personcan be regarded as being present at the same position and a change inpose can be regarded as not occurring in a case where the person is inan upright standstill state or a non-upright standstill state.
 10. Theinformation processing system according to claim 6, wherein thedetection unit detects the person from a first video frame generated ata first time point, the information processing apparatus furthercomprises an estimated position computation unit that estimates aposition of each tracking target person at the first time point usingtracking information that shows a history of information related to thetracking target person detected from the video frame generated beforethe first time point, and an update unit that associates the persondetected from the first video frame with the tracking target person bycomparing a position of each person detected from the first video framein a real world or the image with a position of each tracking targetperson in the real world or the image at the first time point, andupdates the tracking information using a result of the association, andthe update unit estimates the position of the person detected from thefirst video frame in the real world using the estimated body height ofthe person in a case where the person is upright, and estimates theposition of the person detected from the first video frame in the realworld using the temporary-height of the person in a case where theperson is not upright.
 11. A control method executed by a computer, themethod comprising: detecting a person from a video frame; estimating astate of a target person using a result of the detection; and estimatinga body height of the target person on the basis of a height of thetarget person in the video frame in a case where the state of the targetperson satisfies a predetermined condition, wherein the computer, in theestimating a body height of the target person, computes an estimatedbody height of the target person by computing a height of the targetperson in a real world from each of a plurality of video frames in whichthe estimated state satisfies the predetermined condition, andperforming a statistic process on the plurality of computed heights,wherein the computer, in the estimating a state of a target person:computes the height of the target person in the real world from each ofa plurality of video frames in which the target person is included, anddecides a threshold for determining whether or not the person is uprighton the basis of the computed height, for the video frame that issubsequently generated, determines whether or not the target person isupright in the video frame by computing the height of the target personin the real world and comparing the height with the threshold, whereinthe predetermined condition is satisfied in a case where the targetperson is upright, or the predetermined condition is that the targetperson is in a predetermined pose, that likelihood of the target personbeing in the predetermined pose is greater than or equal to a threshold,that a direction of the target person is in a predetermined state, orthat foot of the target person is visible.
 12. A non-transitorycomputer-readable storage medium storing a program causing a computer toexecute the control method of claim 11.